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1. A foreground object and background 
spirite separation and extraction method for 
extracting a foreground object and a background 
spri\te, comprising the steps of: 

obtaining a global motion for transforming 
a coordinate system between a reference frame and a 
frame ^or each of frames in a moving image; 

mapping an original image corresponding to 
said fraVne into a reference coordinate system for 
said eacfi of frames by using said global motion, and 
obtaining\ a pixel value at a point in said reference 
coordinate^ system from pixel values of pixels which 
exist in t\ie same point; 

derating a provisional sprite where 
foreground Objects are deleted; 

cuYting out a first image from said 
provisional ST)rite by using said global motion; 

obtaining a difference image between said 
first image anu said original image; 

extracting a foreground object image as a 
region in said difference image where each 
difference value\ in the region is equal to or higher 
than a threshold,\ and extracting other region as a 
background image; 

mapping said background image to said 
reference coordinate system by using said global 
motion for said eachXof frames by inserting a new 
pixel in a point where a pixel value is not yet 
decided, or by overwrMiing a pixel, for generating 
and outputting said baoJcground sprite. 
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2. The foreground object and background 



sprite separation and extraction method as claimed 
in claim 1, further comprising the steps of: 

\ cutting out a second image from said 
background sprite by using said global motion; 

\obtaining a difference image between said 
second image and said original image; 

extracting a foreground object image as a 
region in said difference image where each 
difference vaVue in the region is equal to or higher 
than a threshol-d. 



3. A foreground object and background 
sprite separation and extraction apparatus for 
extracting a foreground object and a background 
sprite , comprising : \ 

means for \obtaining a global motion for 
transforming a coordinate system between a reference 
frame and a frame for\each of frames in a moving 
image ; \ 

means for mapping an original image 
corresponding to said frame into a reference 
coordinate system for said each of frames by using 
said global motion, and obtaining a pixel value at a 
point in said reference coordinate system from pixel 
values of pixels which exist in the same point; 

means for generating a provisional sprite 
where foreground objects are\ deleted; 

means for cutting aut a first image from 
said provisional sprite by using said global motion; 

means for obtaining a difference image 
between said first image and said original image; 

means for extracting a foreground object 
image as a region in said difference image where 
each difference value in the region is equal to or 



higher -Ghan a threshold, and extracting other region 
as a bac^jLground image; 

means for mapping said background image to 
said reference coordinate system by using said 
global motVlon for said each of frames by inserting a 
new pixel in a point where a pixel value is not yet 
decided, or\ by overwriting a pixel, for generating 
and outputtAng said background sprite. 




Tlie foreground object and background 
sprite sepaffibtion and extraction apparatus as 
claimed in ^eiaim\ 3 , further comprising: 

means fior cutting out a second image from 
said background sprite by using said global motion; 

means for obtaining a difference image 
between said seconoy image and said original image; 

means f or \extracting a foreground object 
image as a region in\said difference image where 
each difference value\ in the region is equal to or 
higher than a threshold. 



5. A computer readable medium storing 
program code for causing a computer to extract a 
foreground object and a baJ^jkground sprite, 
comprising : 

program code mean^ for obtaining a global 
motion for transforming a coordinate system between 
a reference frame and a f rame\ for each of frames in 
a moving image ; 

program code means fbr mapping an original 
image corresponding to said frame into a reference 
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coordinate system for said each of frames by using 
said global motion, and obtaining a pixel value at a 
point in said reference coordinate system from pixel 
values <S(f pixels which exist in the same point; 

program code means for generating a 
provisional sprite where foreground objects are 
deleted; 

urogram code means for cutting out a first 
image from said provisional sprite by using said 
global motion; 

prtogram code means for obtaining a 
difference i^age between said first image and said 
original imac 

program code means for extracting a 
foreground object image as a region in said 
difference imaqe where each difference value in the 
region is equalXto or higher than a threshold, and 
extracting others region as a background image; 

program code means for mapping said 
background image to said reference coordinate system 
by using said glolWl motion for said each of frames 
by inserting/ a newVpixel in a point where a pixel 
value is no'u yet decided, or by overwriting a pixel, 
for generating and outputting said background sprite 
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6. The computebr readable medium as claimed 
30 in claim 5, further comprising; 

program code meWns for cutting out a 
second image from said baqkground sprite by using 
said global motion; 

program code mean^ for obtaining a 
35 difference image between said second image and said 
original image; 

program code means for extracting a 
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foreground object image as a region in said 
differencV image where each difference value in the 
region is >equal to or higher than a threshold. 
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7. A segmentation mask extraction method 
in object codir^g in moving image coding, comprising 
10 the steps of: 

receiv^ing a foreground mask image where a 
foreground part As represented by a first value and 
a background partXis represented by a second value; 

providing a first value as an alpha value 
15 to all shape pixelsX in each of first macro-blocks 

when the number of Aixels of said foreground part in 
said first macro-block is equal to or larger than a 
first predetermined value n (n^l); 

providing saVd first value as said alpha 
20 value to all shape pixels in each of second macro - 
blocks when the number af pixels of said foreground 
part in said second macro-block is equal to or 
larger than a second predetermined value m {m<n) , 
wherein said second macro-plock is close to said 
25 first macro-block where sai^ first value is 
provided; and 

outputting said segirtentation mask. 
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8. The segmentation mas)^ extraction method 
as claimed in claim 7, further comprising the steps 
of: 

receiving each of third ma\?ro-blocks which 
has been determined as said backgroun'd part; and 

providing said first value t^p said third 
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macro-block when a difference image between a 
background image and an original image which 
correspond to said third macro-block includes a 
pixel which has a difference value equal to or 
larger than a threshold. 



10 9. A segmentation mask extraction method 

in object coding ii\ moving image coding, comprising 
the steps of : 

receiving ^ foreground mask image; 
generating \a number map by calculating the 
15 number of pixels of a\ foreground part for each of 
macro-blocks in said foreground mask image; 

initializing\ a foreground map; 
providing a predetermined value to each of 
positions in said foreground map corresponding to 
O 20 first macro -blocks when \a value of said number map 

S=t corresponding to said first macro-block is equal to 

1^ or larger than a first predetermined value n (n^l); 

D ^ providing said predetermined value to each 

of positions in said foreground map corresponding to 
25 second macro-blocks when a Value of said number map 
corresponding to said second macro-block is equal to 
or larger than a second^^^predetermined value m (m<n), 
wherein said second macro^block is close to said 
first macro-block where saia\p^edetermined value is 
30 provided; and 

generating said segmentation mask from 
said foreground map and outputting said segmentation 
mask . 
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\ 10. A segmentation mask extraction 
apparatus in object coding in moving image coding, 
comprising : 

\means for receiving a foreground mask 
image where a foreground part is represented by a 
first value and a background part is represented by 
a second value; 

first macro-block approximation means for 
providing a \f irst value as an alpha value to all 
shape pixels\in each of first macro-blocks when the 
number of pixels of said foreground part in said 
first macro-block is equal to or larger than a first 

predetermined value n (n^l); 

seconp macro-block approximation means for 
providing said lirst value as said alpha value to 
all shape pixels\ in each of second macro-blocks when 
the number of pixels of said foreground part in said 
second macro-blocK is equal to or larger than a 
second predetermined value m (m<n) , wherein said 
second macro-block\ is close to said first macro- 
block where said fiirst value is provided in said 
first macro-block approximation means; and 

means for outputting said segmentation mask. 



11. The segmentation mask extraction 
apparatus as claimed in\ claim 10, further 
comprising: \ 

means for receiving each of third macro- 
blocks which has been determined as said background 
part ; and \ 

means for providing said first value to 
said third macro-block wheA a difference image 
between a background image and an original image 
which correspond to said thind macro-block includes 



a pixel which has a difference value equal to or 
larger than a threshold. 



12. \ A segmentation mask extraction 
apparatus in Abject coding in moving image coding, 
comprising the\steps of: 

meansyfor receiving a foreground mask 
image ; \ 

means for generating a number map by 
calculating the number of pixels of a foreground 
part for each of macro-blocks in said foreground 
mask image ; \ 

means for\ initializing a foreground map; 

means for \providing a predetermined value 
to each of positions \in said foreground map 
corresponding to firsV macro-blocks when a value of 
said number map corresponding to said first macro- 
block is equal to or laVger than a first 
predetermined value n (n^l); 

means for providing said predetermined 
value to each of positionsy in said foreground map 
corresponding to second maoro-blocks when a value of 
said number map corresponding to said second macro- 
block is equal to or larger ^han a second 
predetermined value m (m<n), wherein said second 
macro-block is close to said fVrst macro-block where 
said predetermined value is provided; and 

generating said segmeAtation mask from 
said foreground map and outputting said segmentation 
mask. \ 
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.3. A computer readable medium storing 
program code for causing a computer to extract a 
segmentation mask in object coding in moving image 
coding , comprising : 
5 program code means for receiving a 

foreground mask image where a foreground part is 
represented by a first value and a background part 
is represented by a second valued- 
first macro-block approximation program 
10 code means for providing a first value as an alpha 
value to all shape pixels in each of first macro- 
blocks when the number of pixels of said foreground 
part in said firat macro-block is equal to or larger 
than a first predetermined value n (n^l); 
15 second macro-block approximation program 

code means for providing said first value as said 
alpha value to all shape pixels in each of second 
macro-blocks when tile number of pixels of said 
foreground part in said second macro-block is equal 
20 to or larger than a second predetermined value m 

(m<n) , wherein said second macro-block is close to 
said first macro-bloclA where said first value is 
provided in said first \macro-block approximation 
program code means; an< 
25 program code m^ans for outputting said 

segmentation mask . 
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14. The computer !(^eadable medium as 
claimed in claim 13, further\ comprising ; 

program code means \f or receiving each of 
third macro-blocks which has peen determined as said 
background part; and 

program code means fdoc providing said 
first value to said third macroXblock when a 



difference image between a background image and an 
original \image which correspond to said third macro- 
block includes a pixel which has a difference value 
equal to ot larger than a threshold. 



15. \ A computer readable medium storing 
program code for causing a computer to extract a 
segmentation mask in object coding in moving image 
coding , comprising : 

program code means for receiving a 
foreground mask image; 

programXcode means for generating a number 
map by calculating^ the number of pixels of a 
foreground part for each of macro-blocks in said 
foreground mask imaae; 

program coae means for initializing a 
foreground map; \ 

program code means for providing a 
predetermined value toXeach of positions in said 
foreground map corresponding to first macro-blocks 
when a value of said number map corresponding to 
said first macro-block isy equal to or larger than a 
first predetermined value \n (n^l); 

program code means for providing said 
predetermined value to eachXof positions in said 
foreground map correspondingX to second macro-blocks 
when a value of said number map corresponding to 
said second macro-block is equal to or larger than a 
second predetermined value m (m<n) , wherein said 
second macro-block is close to isaid first macro- 
block where said predetermined value is provided; 
and \ 

program code generatingXsaid segmentation 
mask from said foreground map and outputting said 



segmentati\on mask. 



16 i A segmentation mask extraction method 
for extracting a segmentation mask by using a 
difference image between a background image and an 
image, comprising the steps of: 

obta:Bning said difference image by 
calculating an absolute difference between said 
background image\ and said image for each pixel; 

initializing an energy map for each macro- 
block of said difference image; 

calculating energy values for said each 
macro -block ; \ 

obtaining an average of said energy 
values; \ 

calculating a foreground ratio which is a 
ratio of the size of \a foreground mask to the size 
of said image ; and \ 

generating said segmentation mask by using 
said foreground ratio. \ 



17. The segmentation mask extraction 
method as claimed in claim\l6, further comprising 
the steps of: \ 

obtaining a divided value by dividing said 
energy value by said averageXfor said each macro- 
block, and providing 0 as an energy value to a 
macro-block when said divided Value is equal to or 
smaller than a (oi^l.O); \ 

obtaining a maximum energy value as a 
first predetermined value, setting a second 



predetermined value which Is smaller than said first 
predetermined value, and initializing a foreground 
map ; \ 

initializing a temporary foreground map ; 

providing a predetermined value to each 
macro-block position in said temporary foreground 
map where saiVi energy value is equal to or larger 
than said first predetermined value; 

counting a count number of macro-blocks 
where said temporary foreground map has said 
predetermined value ; 

generating said segmentation mask from 
said foreground map and outputting said segmentation 
mask if a value obtained by dividing said count 
number by the number of all macro-blocks is larger 
than a third predexermined value which is 
predetermined, if nbt, copying values of said 
temporary foreground map to said foreground map; 

iterating k providing step until a divided 
number obtained by ddWiding said count number by the 
number of all macro-blocks becomes larger than said 
third predetermined vaVue, wherein said providing 
step is a step of providing said predetermined value 
to each macro-block position in said temporary 
foreground map where saiu energy value is equal to 
or larger than said second predetermined value, said 
each macro-block being close to a macro-block which 
has said predetermined value in said foreground map; 

when said divided number does not become 
larger than said third predetermined value after 
iterating said providing step, copying values of 
said temporary foreground mapy to said foreground map, 
updating said first predetermined value and said 
second predetermined value, and performing said 
steps after said step of initializing said temporary 
foreground map. \ 
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18. A segmentation mask extraction 
apparatus for extracting a segmentation mask by 
using a difference image between a background image 
and an image, comprising: 

means for obtaining said difference image 
by calculatding an absolute difference between said 
10 background in^age and said image for each pixel; 

meaUs for initializing an energy map for 
each macro-block of said difference image; 

meansv for calculating energy values for 
said each macroAblock; 
15 means ^or obtaining an average of said 

energy values; 

means fdgr calculating a foreground ratio 
which is a ratio ol^ the size of a foreground mask to 
the size of said imJage; and 
20 means for ^generating said segmentation 

mask by using said foreground ratio. 
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19 . The segmen 
method as claimed in clai 
means for obta 



.ation mask extraction 
18 , comprising : 
inlng a divided value by 



dividing said energy value py said average for said 
each macro-block, and providing 0 as an energy value 
to a macro-block when said divided value is equal to 
or smaller than a (a ^1.0); 

means for obtaining a maximum energy value 
as a first predetermined value A setting a second 
predetermined value which is smaVler than said first 
predetermined value, and initialising a foreground 
map ; 
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means for initializing a temporary 
foreground map; 

means for providing a predetermined value 
toXeach macro-block position in said temporary 
foreground map where said energy value is equal to 
or la5tger than said first predetermined value; 

\ means for counting a count number of 
macro-biocks where said temporary foreground map has 
said predetermined value; 

\means for generating said segmentation 
mask from aiaid foreground map and outputting said 
segmentations^ mask if a value obtained by dividing 
said count nmnber by the number of all macro -blocks 
is larger than a third predetermined value which is 
predetermined A if not, copying values of said 
temporary foregVo^i^id map to said foreground map; 

means Vor iterating a providing step until 
a divided number \pbtained by dividing said count 
number by the numD^er of all macro-blocks becomes 
larger than said third predetermined value, wherein 
said providing stepXis a step of providing said 
predetermined value Vo each macro-block position in 
said temporary foreground map where said energy 
value is equal to or rarger than said second 
predetermined value, said each macro-block being 
close to a macro-block wiich has said predetermined 
value in said f oregroundXmap ; 

means for copying values of said temporary 
foreground map to said foreground map, updating said 
first predetermined value and said second 
predetermined value, and performing said steps after 
said step of initializing sai<a temporary foreground 
map, when said divided number Voes not become larger 
than said third predetermined vVlue after iterating 
said providing step. \ 



2(Jy. A computer readable medium storing 
program code\for causing a computer to extract a 
segmentation mask by using a difference image 
between a background image and an image , comprising : 

program code means for obtaining said 
difference imagte by calculating an absolute 
difference between said background image and said 
image for each nixel; 

program code means for initializing an 
energy map for each macro-block of said difference 
image ; \ 

program \code means for calculating energy 
values for said each macro-block; 

program code means for obtaining an 
average of said energy values; 

program code means for calculating a 
foreground ratio which is a ratio of the size of a 
foreground mask to the size of said image; and 

program code\ means for generating said 
segmentation mask by using said foreground ratio. 



21. The computers readable medium as 
claimed in claim 20, comprising: 

program code means for obtaining a divided 
value by dividing said energy value by said average 
for said each macro-block, aAd providing 0 as an 
energy value to a macro-block\when said divided 
value is equal to or smaller than a (a ^1.0); 

program code means for obtaining a maximum 
energy value as a first predetermined value, setting 
a second predetermined value whioh is smaller than 
said first predetermined value, arid initializing a 



foreground map; 

program code means for initializing a 
temporary foreground map; 

prognam code means for providing a 
predetermined value to each macro-block position in 
said temporary foreground map where said energy 
value is equal t© or larger than said first 
predetermined valjpe; 

program\code means for counting a count 
number of macro-blocks where said temporary 
foreground map has \said predetermined value; 

program code means for generating said 
segmentation mask from said foreground map and 
outputting said segmentation mask if a value 
obtained by dividing said count number by the number 
of all macro-blocks is\ larger than a third 
predetermined value whAch is predetermined, if not, 
copying values of said Tiemporary foreground map to 
said foreground map; \ 

program code maans for iterating a 
providing step until a divided number obtained by 
dividing said count number\ by the number of all 
macro-blocks becomes largert than said third 
predetermined value, wherein said providing step is 
a step of providing said predetermined value to each 
macro-block position in said\temporary foreground 
map where said energy value is equal to or larger 
than said second predetermined^ value , said each 
macro-block being close to a mapro-block which has 
said predetermined value in said foreground map; 

program code means for\copying values of 
said temporary foreground map to \said foreground map, 
updating said first predetermined Value and said 
second predetermined value, and performing said 
steps after said step of initializiira said temporary 
foreground map, when said divided number does not 
become larger than said third predetermined value 
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after iterating said providing step 
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12. A segmentation mask extraction method 
for extracting a segmentation mask by using a 
difference V^^S® between a background image and an 
image, comprising: 
10 a first step of regarding each of first 

macro-blocks as the foreground when an energy value 
of said first \macro-block which is obtained by said 
difference image is equal to or larger than a first 
predetermined ^\alue ; 
15 a second step of regarding each of second 

macro-blocks as V:he foreground when an energy value 
of said second mstcro-block is equal to or larger 
than a second predetermined value, said second 
macro-block being Vclose to a macro-block which is 
20 determined as the foreground in said first step. 



25 23. The segmentation mask extraction 

method as claimed in c\Laim 22, further comprising a 
step of iterating said\second step for predetermined 
times . 
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24. A segmentation mask extraction method 
for extracting a segmentation mask by using a 
3 5 difference image between a ba\:kground image and an 
image, comprising the steps of> 

calculating energy values of each macro- 
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block from \said difference image and calculating an 
average of said energy values; 

obtaining a divided value by dividing said 
energy value \by said average for said each macro- 
block, and providing 0 as an energy value to a 
macro-block wWen said divided value is equal to or 
smaller than a ^predetermined value; 

regarding each of first macro-blocks as 
the foreground when said energy value of said first 
macro-block is equal to or larger than a first 
predetermined value ; 

iteratin\g, predetermined times, a step of 
regarding each of second macro-blocks as the 
foreground when saiVi energy value of said second 
macro-block is equal to or larger than a second 
predetermined value ,\ said second macro-block being 
close to said first iriacro-block which is determined 
as the foreground. 
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25. A segmentaVtion mask extraction 
apparatus for extracting\a segmentation mask by 
25 using a difference image/ petween a background image 
and an image, comprisir^g : 

first means for Regarding each of first 
macro -blocks as the foreground when an energy value 
of said first macro-block which is obtained by said 
30 difference image is equal tc\ or larger than a first 
predetermined value ; 

second means for regarding each of second 
macro-blocks as the foreground when an energy value 
of said second macro-block is \eqlial to or larger 
35 than a second predetermined value, said second 

macro-block being close to a macro-block which is 
determined as the foreground in ^said first means. 



26. VThe segmentation mask extraction 
apparatus as claimed in claim 25, further comprising 
means for iteratiing the process by said second means 
for predetermined times. 



27, A segmentation mask extraction 
apparatus for extracting a segmentation mask by 
using a differenae a.mage between a background image 
and an image, comprising: 

means for calculating energy values of 
each macro-block from said difference image and 
calculating an average of said energy values; 

means for obtaining a divided value by 
dividing said energy v^lue by said average for said 
each macro-block, and Aroviding 0 as said energy 
value to a macro-block wien said divided value is 
equal to or smaller tharaXa predetermined value; 

means for regarding each of first macro- 
blocks as the foreground Wnen said energy value of 
said first macro-block is^Ajual to or larger than a 
first predetermined value ;\\ 

means for iteratAna, predetermined times, 
a step of regarding each of\s(BCond macro-blocks as 
the foreground when said eneVgys value of said second 
macro-block is equal to or larger than a second 
predetermined value, said second macro-block being 
close to said first macro-block which is determined 
as the foreground. \ 
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281 A computer readable medium storing 
program code \f or causing a computer to extract a 
segmentation mask by using a difference image 
between a background image and an image, comprising: 

first program code means for regarding 
each of first \macro-blocks as the foreground when an 
energy value o\f said first macro-block which is 
10 obtained by said difference image is equal to or 
larger than a first predetermined value; 

seconp program code means for regarding 
each of second macro-blocks as the foreground when 
an energy value \of said second macro-block is equal 
15 to or larger than a second predetermined value, said 
second macro-bloffik being close to a macro-block 
which is determined as the foreground in said first 
program code meanJ 



20 



25 



29. The c^pmputer readable medium as 
claimed in claim 28, \further comprising program code 
means for iterating tne process by said second 
program code means for\ predetermined times. 
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30. A computer Readable medium storing 
program code for causing a computer to extract a 
segmentation mask by using\a difference image 
between a background image )and an image, comprising; 

program code means for calculating energy 
values of each macro-block fi^om said difference 
image and calculating an average of said energy 
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values ; 

program code means for obtaining a divided 
value by dividing said energy value by said average 
for said each macro-block, and providing 0 as said 
energy value to a macro -block when said divided 
value is equal to\or smaller than a predetermined 
value ; 

program ^ode means for regarding each of 
first macro-blocks as the foreground when said 
energy value of said first macro-block is equal to 
or larger than a fir^t predetermined value; 

program codte means for iterating, 
predetermined times, a step of regarding each of 
second macro-blocks as\the foreground when said 
energy value of said se\pond macro-block is equal to 
or larger than a second\predetermined value, said 
second macro-block being\ close to said first macro- 
block which is determined as the foreground. 
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